Parallelize task instance gather and support --gather on Modal#199
Merged
AlienKevin merged 40 commits intoSWE-bench:mainfrom Mar 9, 2026
Merged
Parallelize task instance gather and support --gather on Modal#199AlienKevin merged 40 commits intoSWE-bench:mainfrom
AlienKevin merged 40 commits intoSWE-bench:mainfrom
Conversation
…t take longer to gather
Previously, the script would fail if `git commit` was attempted with no changes. This was observed in cases like `Automattic__mongoose.5f57a5bb` where the applied patch resulted in no tracked changes. Now, we check `git status --porcelain` before committing and skip the instance if no changes are detected.
for more information, see https://pre-commit.ci
Codecov Report✅ All modified and coverable lines are covered by tests.
🚀 New features to boost your workflow:
|
…ateless git ls-remote
- Switch from per-task clones to per-worker persistent repositories. - Reduces clone operations from O(tasks) to O(workers) (e.g. 1400 -> 17). - Eliminates file locking race conditions. - Total gather time for Javascript is now ~5 minutes (bottlenecked by math.js).
AlienKevin
force-pushed
the
kevin/bug-gen-gather
branch
from
8cefa3c to
e42a5e2
Compare
AlienKevin
force-pushed
the
kevin/bug-gen-gather
branch
from
98898a8 to
302b73e
Compare
for more information, see https://pre-commit.ci
AlienKevin
force-pushed
the
kevin/bug-gen-gather
branch
from
52ba582 to
f7f68cb
Compare
for more information, see https://pre-commit.ci
Cause: - gather invoked apply commands with a relative patch path (`../logs/run_validation/.../patch.diff`). - During modal gather, each worker runs from a temporary repo directory under `/tmp/...`, so that relative path did not resolve to the mounted logs directory. - `git apply` and fallback `patch` both failed with "can't open patch ... No such file or directory", resulting in dropped instances and empty/underfilled task outputs. Fix: - Resolve `patch.diff` to an absolute path before apply. - Shell-quote that absolute path and pass it to every command in `GIT_APPLY_CMDS`. Result: - Patch application no longer depends on worker cwd; gather can apply valid rust patches and produce task instances consistently.
Root cause: upload_tasks_to_hf_modal.py was hardcoded to javascript paths in both task discovery and per-repo processing. Running with --language rust still read /data/javascript/... and javascript/task_insts, which breaks Rust upload workflows and can surface as missing/empty problem statements for non-JS datasets.
Fix: thread a language argument through the local entrypoint and worker function, list files from {language}/task_insts, and pass language explicitly through process_repo.map so each worker reads /data/{language}/task_insts and /data/{language}/issue_gen.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
# Conflicts: # scripts/bug_gen_modal.py
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces parallel processing to the task instance gathering phase to significantly improve performance for large datasets and adds support for the gather phase in the Modal workflow.
Key Changes:
Before this PR, large repos like math.js with >800 task instances timed out after 20 minutes due to slow, sequential git branch creation and git push. After this PR, large repos finish in minutes.
ProcessPoolExecutorto process task instances in parallel, utilizing multiple cores.repo_name_pid_subfolder) to prevent race conditions during concurrent Git operations.--gatherCLI flag (skipping generation/validation).Question: do we want to fix the FAIL_TO_PASS to PASS_TO_FAIL?:
swe-smith currently uses
FAIL_TO_PASSfor tests that pass before the bug patch but fails afterwards, which inverts the semantic and causes confusion. A more intuitive name would bePASS_TO_FAILso I used this convention in this PR. However, if we are to adopt this new convention, the rest of the code and datasets need to be updated, so I'm not sure whether it's worth it?Resolution: Flip
PASS_TO_FAILtoFAIL_TO_PASSin alignment with SWE-bench naming convention when outputing the task instance jsons.Test command